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SUMMARY 


This report has been prepared to closeout a NASA grant to Mississippi State University (MSU) 
for research into situation awareness (SA) and automation in the advanced commercial aircraft 
cockpit. The grant was divided into two obligations including $60,000 for the period from May 
11, 2000 to December 25, 2000. The information presented in this report summarizes work 
completed through this obligation. It also details work to be completed with the balance of the 
current obligation and unobligated funds amounting to $50,043, which are to be granted to North 
Carolina State University for completion of the research project from July 31, 2000 to May 10, 
2001. 

This research was to involve investigation of a broad spectrum of degrees of automation of 
complex systems on human-machine performance and SA. The work was to empirically assess 
the effect of theoretical levels of automation (LOAs) described in a taxonomy developed by 
Endsley & Kaber (1999) on naive and experienced subject performance and SA in simulated 
flight tasks. The study was to be conducted in the context of a realistic simulation of aircraft 
flight control. The objective of this work was to identify LOAs that effectively integrate humans 
and machines under normal operating conditions and failure modes. In general, the work was to 
provide insight into the design of automation in the commercial aircraft cockpit. Both laboratory 
and field investigations were to be conducted. 

At this point in time, a high-fidelity flight simulator of the McDonald Douglas (MD) 1 1 
aircraft has been completed. The simulator integrates a reconfigurable flight simulator developed 
by the Georgia Institute of Technology and stand-alone simulations of MD-1 1 auto flight systems 
developed at MSU. Use of the simulator has been integrated into a study plan for the laboratory 
research and it is expected that the simulator will also be used in the field study with actual 
commercial pilots. In addition to the flight simulator, an electronic version of the Situation 
Awareness Global Assessment Technique (SAGAT) has been completed for measuring 
commercial pilot SA in flight tasks. The SAGAT is to be used in both the lab and field studies. 
Finally, the lab study has been designed and subjects have been recruited for participation in 
experiments. This study will investigate the effects of five levels of automation, described in 
Endsley & Kaber’ s (1999) taxonomy and applied to the MD-11 autoflight system, on private 
pilot performance and SA in basic flight tasks by using the MD-11 simulator. The field study 
remains to be planned and executed. Experimental test scenarios need to be developed to assess 
commercial pilot ability to effectively interact with cockpit automation in conducting routine 
flight operations and dealing with emergency conditions. As well, a software utility for analyzing 
the SAGAT data collected during each study has yet to be developed. 

These research activities were to be complemented by a survey of the state-of-the-art of 
human-centered automation research. Preparations have been made for a conference including 
calling for technical papers, reviewing abstracts, publishing a conference brochure and 
registration packet, scheduling conference presentations, and publishing a preliminary program. 
The conference is to be held this October 15-19 and a proceedings is to be compiled as part of 
this research program. 



SITUATION AWARENESS & LEVELS OF AUTOMATION: EMPIRICAL 
ASSESSMENT OF LEVELS OF AUTOMATION IN THE COMMERCIAL COCKPIT 


Project Accomplishments: 

Virtual Flight Simulator Development 

During the past 2.5 months, the virtual flight simulator software was finalized. Microsoft 
Windows communications programming was completed to facilitate altitude and pitch control of 
the McDonald Douglas (MD) 11 aircraft, as presented through the Georgia Institute of 
Technology (GIT) Reconfigurable Flight Simulator (RFS). This was accomplished by 
developing and integrating a simulation of the MD-1 1 Multi-Control Display Unit (MCDU) with 
the RFS. The MCDU simulation was prototyped using E-Sim’s RAPID© software. Other MD- 
1 1 manual flight controls and autoflight system components were also simulated using RAPID©, 
including thrust and aileron controls, the Flight Mode Annunciator (FMA) and the Flight Control 
Panel (FCP). These soft prototypes of flight displays and controls as part of the MD-1 1 cockpit 
were all integrated with the GIT RFS. 

In order to simulate different modes of automation in the MD-11 cockpit, the RAPID© 
prototypes and the RFS have been combined in various ways. The five levels of automation 
(LOAs) described in Endsley & Kaber’s (1999) taxonomy, which have been applied to the MD- 
11 cockpit and simulated through the virtual flight simulator include: (1) Batch Processing — 
This LOA resembles speed, heading and altitude control modes of the MD-11; (2) “Shared 
Control” - This LOA resembles switching between MCDU control and use of the FCP to set 
flight parameters, and monitoring control modes and current flight parameter settings through the 
FMA; (3) “Decision Support” - This LOA resembles using the MCDU to program a flight plan; 

(4) “Supervisory Control” - This LOA resembles full automation of flight control tasks with the 
capability for a pilot to intervene in automated processing and reprogram a new flight plan; and 

(5) “Full Automation” - This LOA represents completely automated control of the aircraft. 
Under Batch Processing the simulator presents the MCDU, the FMA, the Primary Flight Display 
(PFD), and the Navigational (NAV) Display. Shared Control involves the presentation of all 
simulator components including the FCP and throttle and attitude controls. Under Decision 
Support, Supervisory Control and Full automation, the simulator presents the same displays as 
presented in the Batch Processing mode. The manner in which the simulator interfaces receive 
and respond to input under different LOAs resembles the behavior of real MD-11 interface 
controls, however, personal computer peripherals and devices are used for input purposes. 

Finally, a database of information on international airspace was developed to allow realistic 
flight scenarios to be re-created for testing the virtual flight simulator and user ability to fly the 
MD-11 simulator according to specific flight plans. Currently the database contains waypoint 
and route information for the Gulf of Mexico and Colombia, South America. It includes flight 
constraint information, such as minimum altitudes at waypoints in order to avoid terrain, 
maximum air speeds on particular routes, and approach paths to airports. This data can be 
presented to simulator users for flight path development or the database can be linked to the 
MCDU component of the simulator to allow users to select from predefined flight paths included 
in the database. 


Simulator Control and Situation Awareness M easurement Software 


In addition to the virtual flight simulator development work, a software application has been 
programmed to manage the arrangement of displays as part of the simulator on computer 
monitors. A “parent” process was created to position an out-of-cockpit view, the PFD, NAV 
Display, MCDU, FMA and FCP across two high-resolution graphics monitors. The process 
arranges the displays of the flight simulator in a manner similar to their arrangement in a real 
MD-11 cockpit. Beyond this, the parent process integrates the virtual flight simulator with the 
Commercial Pilot Situation Awareness Global Assessment Technique (SAGAT) software 
developed by SA Technologies. The SAGAT package presents queries on pilot perceptual 
knowledge, comprehension of the state of an aircraft, and projections of future aircraft states for 
different phases of flight, including take-off, en route and landing. The package is used to 
develop a “picture” of pilot SA at specific points in time during a flight. The parent process 
manages the execution of the virtual flight simulator and the SAGAT software. It has the 
capability to halt or freeze the flight simulator at any point in time during a simulated flight and 
to present the SAGAT application to a user. The parent process is capable of delivering SAGAT 
queries at specific points in time during a flight or when the aircraft reaches specific waypoints 
in a flight plan. 

The SAGAT application includes approximately 45 queries relating to the levels of SA 
defined by Endsley (1995), including “Level 1” SA - perception; “Level 2” SA - 
comprehension; and “Level 3” SA — projection. The application includes 20 queries concerning 
Level 1 SA, 16 queries concerning Level 2 SA and 9 queries on Level 3 SA. The application 
allows an experimenter to select a phase of aircraft flight to be studied and to select specific 
queries for presentation to simulator pilots. The queries include closed and open-ended questions 
requiring pilots to specify, for example, flight parameter settings (heading, altitude, air speed, 
etc.). The application also allows an experimenter to define the number of queries to be posed to 
a simulator user and the amount of time available for a pilot to respond to them. Or, the 
experimenter may allow an unlimited response time. Finally, the application stores pilot 
responses to queries for later analysis by comparison with actual aircraft states recorded during a 
flight simulation. The application is very powerful for describing a pilot s state of awareness 
during flight as well as errors in SA that could impact flight task performance. 

With respect to the integration of the SAGAT application and the virtual flight simulator, 
when the parent process freezes the flight simulator in order to execute the SAGAT application, 
the process minimizes the simulator displays and opens the SAGAT application screen. A user is 
not permitted to view the simulator displays in responding to the SAGAT queries. Once all 
queries have been answered, the parent process minimizes the SAGAT application screen, opens 
the flight simulator displays and resumes the simulation from the point at which it was frozen 
prior to the execution of the SAGAT application. 

l aboratory Study Plan and Subject Recruiting 

A study plan has been formulated to assess the effects of the MD-11 modes of automation 
currently simulated through the virtual flight simulator on naive subject performance, SA and 
mental workload in flight control tasks. The experiment is to identify LOAs that optimally 
integrate human and automation in flight tasks. Specifically, the study is to demonstrate LOAs 


that maximize performance under normal operating conditions and failure modes and minimize 
decrements to SA. 

A large sample of private pilots from the Starkville, Mississippi area has been recruited for 
participation in the experiment. The subjects will be required to fly the MD-11 flight simulator 
through a recreation of the situation encountered by the pilots of the American Airlines (AA) 
Flight 965, which crashed near Cali, Colombia in 1995. The experimental scenario will involve 
en-route flight tasks and approach programming, as well as an approach revision due to a runway 
change in flight. The en-route tasks will include monitoring simulated MD-1 1 autoflight systems, 
flying the simulator under manual control, or using the aircraft FCP or FMS for programming 
flight paths. Subjects will be expected to deal with the approach revision under high time 
pressure (near the estimated time of landing). The approach revision requirement will be 
exploited to assess the impact of LOAs on human recovery of the aircraft from an erroneous 
flight path in terms of programming performance, SA and workload. 

During the experiment, performance data, including time-to-flight-task completion and flight 
plan programming errors will be recorded. Using the SAGAT for commercial piloting software, 
we will assess SA. The virtual flight simulator will be frozen at random points in time during 
experiment trials and subjects will be asked to respond to SA queries intended to assess their 
knowledge at perceptual, comprehension and projection levels. The percentage of correct 
responses to these queries will be recorded as a measure of their level of SA. Mental workload 
will be assessed using the NASA-Task Load Index (TLX). 

All experimental testing is expected to be complete by the close of August 2000 and data 
analysis will be performed during the first half of the Fall term. Effects of LOAs on the various 
response measures will be tested using multivariate and univariate analyses of variance. Any 
significant effects will be further investigated using conservative multiple comparison 
procedures including Tukey’s Honestly Significant Difference tests. Results will be explained in 
terms of the allocation of flight control functions to pilots and automation under the various 
LOAs. 

Field Study Plan 

The laboratory study is to be replicated in a field situation using alternate flight scenarios and 
error conditions, as well as experienced commercial pilots. The objective of this experiment is to 
gain a more realistic understanding of the effects of actual and theoretical modes of advanced 
commercial aircraft automation on pilot performance, SA and cognitive workload. The virtual 
flight simulator and supporting systems will be transported to Atlanta, Georgia and setup in a 
secure facility adjacent to Atlanta/Hartsfield International Airport and the Delta Airlines training 
facility. 

A large sample of Airline Pilots Association members is to be recruited from the Atlanta, 
Georgia region. They will be required to perform en-route and approach flight tasks using the 
virtual flight simulator under various LOAs selected from Endsley & Kaber’s (1999) taxonomy. 
During their experiences with the simulator, pilot accuracy and speed in flight tasks will be 
recorded along with SA using the SAGAT for commercial piloting. As in the lab study, we will 
also subjectively assess mental workload using the NASA-TLX or Modified Cooper-Harper 
mental workload rating scale depending upon the workload results of the first experiment. 

At this point in time, SA Technologies, the Subcontractor, is developing experimental test 
scenarios for use in the field study. The key constraint in this task is that commercial airline 



pilots are usually familiar with major aviation incidents, including the AA Flight 965 incident, its 
causes, and the consequences. Therefore, recreating this type of scenario for testing with the 
virtual flight simulator would be ineffective in terms of attempting to develop a realistic 
“picture” of pilot performance capabilities and SA under novel error conditions because of pilot 
a priori knowledge of the accident circumstances and potential safety control measures. 

Subsequent to this task, the Subcontractor will develop a relational database and custom 
spreadsheet to accurately analyze the SA data recorded during both the lab and field study using 
the SAGAT software package. The database will be used to store actual naive subject and pilot 
responses to SA queries administered during experimental trials. It will also contain information 
on the “real” state of the virtual flight simulator at the times queries that were administered. The 
database will allow for comparison of the system data with subject perceptions of simulator 
states. The relational database will facilitate analysis of subject responses to open-ended 
questions and will allow for grades of accuracy to be established for responses to queries. This 
work is to occur through an additional obligation of NASA funds in Spring 2001 . 

Survey of the State-of-the-Art in Human- Automation Interaction 

A conference on human performance, SA and automation has been planned and organized 
through support as part of this research grant. The conference is to be held this October 15-19 in 
Savannah, Georgia. At the time of this report, 60 technical papers, 5 panel submissions and over 
20 technical posters had been reviewed and accepted for presentation at the meeting. Through 
NASA funding, a conference brochure, registration packet and preliminary program have all 
been published and distributed to potential attendees. It is anticipated that over 200 of the leading 
researchers in human-automation interaction and SA will attend the conference. 

The meeting is intended to develop a synergy of research results on theoretical work in SA 
assessment and applied studies of human-automation interaction. A conference proceedings 
(research survey) will be edited and published through NASA support committed as part of this 
research award. It is expected that papers included in the proceedings will present successful 
approaches to system design to facilitate high levels of operator SA in conjunction with 
automation. The book will provide readers with information on integrated approaches to system 
design considering automation and information systems technology and human SA to optimize 
human/system performance. The survey will serve to advance research in directions similar to, 
and including, that pursued as part of this project. 

Deliverables: 


This technical report and the development of a research World Wide Web site were identified as 
the key deliverables of this project in the research proposal. A web site presenting all the 
information developed as part of this research project to date has been published at. 
http://www.ie.msstate.edu/people/faculty/kaber/nasa/index.html. The site presents reviews of the 
Delta Pilot’s Reference Manual, hierarchical analyses of the functions of MD-11 autoflight 
systems, information on the virtual flight simulator including the RFS and prototypes of the MD- 
1 1 MCDU, FCP, and FMA developed using RAPID©. It also presents information on the 
research team involved in the project activities, including three Mississippi State University 
graduate students. Finally, the site provides virtual tours of the MD-1 1 cockpit and the aircraft 
itself. 



Other Research Outcomes 


At this point in time the project has generated five publications, including those listed in the final 
report for Grant No. NCC1330 99050488 and an additional archival journal article. The current 
reference for the latter publication is: 

Kaber, D. B., Tan, K-W. and Riley, J. R. (in review). Improved usability of aviation automation 
through direct manipulation and graphical user interface design. Submitted to the Internationa l 
Journal of Aviation Psychology - 
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